Heuristic Data Driven Decision Making and Data Story Telling
In a fast paced environment, decisions are often made quickly and often based on gut decisions. Progressive companies have tried as much as possible to leverage big data and analytics to be data driven organizations. Ideally, leveraging data would help to override biases and often gut instincts and ulterior motives that may stand behind a scientific hypothesis or business question. One of the many things we have learned from behavioral economics is that humans tend to over interpret data into unreliable patterns that lead to incorrect conclusions. Francis Bacon recognized this over 400 years ago:
"the human understanding is of its own nature prone to suppose the existence of more order and regularity in the world than it finds"
Anyone can tell a story with data. And with lots of data a good data story teller can tell a story to support any decision they want, good or bad. Decision makers can be easily duped by big data, ML, AI, and various BI tools into thinking that their data is speaking to them. As Jim Manzi and Stefan Thomke state in Harvard Business Review in the absence of experimentation and causal inference
"executives end up misinterpreting statistical noise as causation—and making bad decisions"
Data seldom speaks, and when it does it is often lying. This is the impetus behind the introduction of what became the scientific method. The true art and science of data science is teasing out the truth, or what version of truth can be found in the story being told. I think this is where experimentation and causal inference are most powerful and create the greatest value in the data science space. John List and Uri Gneezy discuss this in their book 'The Why Axis.'
"Big data is important, but it also suffers from big problems. The underlying approach relies heavily on correlations, not causality. As David Brooks has noted, 'A zillion things can correlate with each other depending on how you structure of the data and what you compare....because our work focuses on field experiments to infer causal relationships, and because we think hard about these causal relationships of interest before generating the data we go well beyond what big data could ever deliver."
Decision Making Under Uncertainty, Risk Aversion, and The Dunning-Kruger Effect
Kahneman (in Thinking Fast and Slow) makes an interesting observation in relation to managerial decision making. Very often managers reward peddlers of even dangerously misleading information (data charlatans) while disregarding or even punishing merchants of truth. Confidence in a decision is often based more on the coherence of a story than the quality of information that supports it. Those that take risks based on bad information, when it works out, are often rewarded. To quote Kahneman:
"a few lucky gambles can crown a reckless leader with a Halo of prescience and boldness"
The essence of good decision science it understanding and seriously recognizing risk and uncertainty. As Kahneman discusses in Thinking Fast and Slow, those that often take the biggest risks are not necessarily any less risk averse, they simply are often less aware of the risks they are actually taking. This leads to overconfidence and lack of appreciation for uncertainty, and a culture where a solution based on pretended knowledge is often preferred and even rewarded. Its easy to see how the Dunning-Kruger effect would dominate. This feeds a viscous cycle that leads to collective blindness toward risk and uncertainty. It leads to taking risks that should be avoided in many cases, and prevents others from considering smarter calculated risks. Thinking through an experimental design (engaging Kahneman's system 2) provides a structured way of thinking about business problems and all the ways our biases and the data can fool us.. In this way experimentation and causal inference can ensure a better informed risk appetite to support decision making.
Just as rapid cycles of experiments in a business setting can aid in the struggle with the knowledge problem, experimentation and causal inference can aid us in our struggles with biased decision making and biased data. Data alone doesn't make good decisions because good decisions require something outside the data. Good decision science leverages experimentation and causal inference that brings theory and subject matter expertise together with data so we can make better informed business decisions in the face of our own biases and the biases in data.
A business culture that supports risk taking coupled with experimentation and causal inference will come to value a preferred solution over pretended knowledge. That's valuable.
See also:
No comments:
Post a Comment